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ABSTRACT 

An  innovative  approach  for  navigation  in  non-GPS  environment  is  presented  based  on  all  source  adaptive  fusion  of  any 
available  information  encompassing  passive  imaging  data,  digital  elevation  terrain  data,  IMU/GPS,  altimeters,  and  star 
tracker.  The  approach  provides  continuous  navigation  through  non-GPS  environment  and  yields  an  improved  navigation 
in  the  presence  of  GPS.  The  approach  also  provides  reduced  target  location  error  and  moving  target  indication. 

Keywords:  Optical  flow,  GPS,  target  location  error,  structure  from  motion 

1.  BACKGROUND 

There  is  a  critical  need  to  provide  fully  autonomous  robust  navigation  capability  in  GPS  jamming  or  signal  interruption 
environment.  The  US  Air  force,  Navy,  and  Army  are  keenly  interested  in  such  capability  to  reduce  the  vulnerability  of 
GPS  navigation  that  can  be  to  either  deliberately  or  unintentionally  interrupted.  The  field  of  computer  vision  has 
witnessed  many  approaches  for  computing  ego-motion  which  estimates  an  observer’s  movement  (sensor  or  biological 
organism)  from  optical  flow  measurements.  Optical  flow  is  defined  as  “the  apparent  motion  of  the  brightness  patterns”. 
The  optical  flow  is  characterized  by  a  field  of  2-D  velocity  vectors  which  are  the  projections  of  the  3-D  velocity  vectors 
of  the  surface  points  onto  the  image  plane.  The  2-D  velocity  vectors  are  derived  from  sequences  of  images.  The 
measurement  of  the  2-D  image  velocity  at  a  given  point  can  be  used  to  infer  the  3-D  velocity  of  the  imaging  platform. 
Since  the  optical  flow  is  a  projection  of  a  3-D  velocity  vector  onto  a  2-D  image  plane,  there  is  an  inherent  ambiguity  in 
inferring  the  3-D  velocity.  The  observed  optical  flow  therefore  needs  to  be  fused  with  other  measurements  such  as  depth 
to  remove  the  ambiguity  and  to  provide  an  estimate  of  the  3-D  velocity  vector  of  the  imaging  platform.  Knowing  the 
observer  velocity  vector  and  the  last  GPS  position  estimate  will  enable,  in  theory,  navigation  in  non-GPS  environment. 
Using  only  optical  flow  measurements  for  navigation  in  non-GPS  environment  encounters  difficulties  due  to  the 
variation  of  depth  across  the  image  field  and  the  presence  of  noise  in  the  sequence  of  images.  Other  approaches  combine 
precision  radar  sensors  with  digital  terrain  elevation  databases  to  serve  as  a  back-up  to  the  GPS  system.  Such 
approaches  require  high-level  of  accuracies  of  digital  terrain  data  and  radar  sensor. 

This  paper,  describes  an  innovative  approach  developed  by  Northrop  Grumman  and  AFRL  Munitions  directorate  to 
adaptively  fuse  in  real-time  all  available  navigation  data  such  as  IMU/GPS,  altimeters,  star  tracker,  passive  imaging 
sensor,  and  digital  elevation  database.  The  integration  of  passive  imaging  sensors  has  some  important  advantages. 
Foremost,  the  sensors  are  completely  passive,  and  can  operate  in  an  environment  where  the  GPS  signal  may  be  difficult 
to  receive.  Secondly,  the  sensors  are  immune  to  disruptions  in  the  radio  spectrum.  The  proposed  approach  has  been 
implemented  and  demonstrated  high  quality  navigation  performance.  An  overview  of  ASAF  is  in  section  2,  and  its 
performance  is  shown  in  section  3.  Conclusions  are  made  in  section  4. 

2.  ALL  SOURCE  ADAPTIVE  FUSION 

The  All  Source  Adaptive  Fusion  (ASAF)  approach  developed  by  Northrop  Grumman  and  AFRL  Munitions  director 
performs  at  levels  which  are  difficult  to  attain  with  single  non-integrated  navigation  sources.  Figure  1  depicts  various 
applications  of  the  ASAF.  ASAF  is  a  novel  approach  which  offers  the  following  principle  advantages: 
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1 .  Has  a  modular  software  solution  which  can  be  easily  integrated  on  existing  and  future  platforms; 

2.  Combines  in  real-time  various  sources  of  information  as  they  become  available  (adaptive  to  what  is  available); 

3.  Reduces  targeting  error  for  weapon  systems  using  EO/IR  imaging  sensors  and  provides  more  accurate  sensor 
position  and  velocity  critical  to  the  coherent  processing  of  the  Synthetic  Aperture  Radar  (SAR)  Sensors; 

4.  Provides  passive  moving  target  indication  (MTI);  and 

5.  Supports  both  structure-from-motion  (SFM)  and  motion-from- structure  (MFS),  thus  providing  navigation 
aiding  in  the  GPS-denied  environment  as  well  as  obstacle  avoidance  and  3-D  digital  terrain  mapping  in  the 
presence  of  GPS.  The  3-D  digital  map  can  be  generated  using  ASAF  software  hosted  on  multiple  surveillance 
UAVs  flying  over  urban  areas  to  provide  routing  support  for  other  platforms  and  3-D  mapping  for  ground  troop 
situation  awareness. 
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Figure  1:  ASAF  has  wide  range  of  applications 
The  major  components  of  the  ASAF  are  described  in  the  following  paragraphs. 

2.1  Optical  flow  processing 

Optical  flow  (OF)  is  the  computation  of  pixel  displacement  or  displacement  velocity  given  a  sequence  of  images.  There 
are  three  main-stream  approaches  to  this  problem  ([4]):  differential,  block  matching  and  frequency  methods.  The 
differential  method  was  based  on  the  assumption  that  brightness  is  conserved  as  pixels  propagate.  This  constraint  leads 
to  the  normal  flow  equation,  which  relates  the  brightness  gradient  vector  in  2D  image  space  and  brightness  temporal 
changes  in  time.  This  equation  cannot  generate  a  unique  solution  and  therefore  needs  a  second  constraint.  Horn  and 
Schunck  built  the  foundation  of  the  differential  method  by  assuming  optical  flow  field  to  change  smoothly  over  the 
entire  image  ([1]).  This  assumption  does  not  hold  along  occlusion  boundaries  and  many  authors  have  contributed  to 
overcome  this  shortcoming.  The  variational  method  and  its  sibling  modification  ([3])  replace  the  smoothing  constraint 
function  from  an  identity  function  by  a  nonlinear  function  which  suppresses  smoothing  on  region  where  flow  gradient  is 
large.  In  other  words,  the  weight  of  smoothing  constraint  is  not  uniformly  distributed  in  the  entire  image,  but  intrinsically 
adjusted  as  part  of  the  optimization  process.  The  advantage  of  the  differential  method  is  that  it  produces  a  dense  flow 
field. 
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The  frequency  method  showed  high  accuracy  but  its  computation  is  expensive  ([4]).  In  the  block  matching  method, 
displacement  of  an  image  block  is  computed  based  on  the  best  match  over  a  search  neighborhood  using  a  correlation 
metric.  Since  checking  each  element  in  the  search  neighborhood  is  slow,  Lucas  and  Kanade  proposed  a  steepest  descent 
approach  to  speed  up  this  process  ([2]).  This,  together  with  feature  selection  and  monitor  ([5]),  has  formed  the  popular 
feature  tracking  algorithm  referred  to  as  Kanade-LucasTomasi  (KLT). 

We  have  selected  the  KLT  algorithm  among  a  suite  of  optical  flow  algorithms  to  implement  due  to  three  reasons.  First, 
since  we  require  only  a  limited  number  of  features  for  the  subsequent  fusion  process  (secion  2.3),  knowledge  of  entire 
flow  field  provided  by  differential  method  is  not  needed.  Secondly,  performance  evaluation  literature  concluded  KLT  is 
competitive  in  terms  of  accuracy  and  processing  speed  ([4]).  Thirdly,  experimental  robots  have  been  built  using  this 
method  after  independent  evaluation  of  other  approaches  ([6]). 

There  are,  however,  technical  challenges  that  KLT  has  not  handled.  First,  as  pointed  out  by  other  authors  as  well, 
accuracy  of  KLT  tracking  is  sensitive  to  initial  guess  of  displacement.  This  is  due  to  the  nature  of  steepest  descent  which 
tends  to  seek  a  nearby  minimum  point,  regardless  of  whether  it  is  a  local  or  global  minimum  within  the  search  region. 
For  highly  repeating  image  texture,  KLT  can  be  easily  trapped  by  a  local  minimum  point.  Second,  large  displacement  is 
handled  by  constructing  an  image  pyramid  and  applying  KLT  tracking  in  each  layer  using  the  displacement  result  of 
upper  layer  as  the  initial  guess.  If  the  variation  of  feature  displacement  is  substantial,  successive  initial  guess  by  pyramid 
scheme  will  break  down.  Third,  the  feature  scaling  problem  is  not  addressed.  This  problem  may  be  resolved  by  the  Scale 
Invariant  Feature  Tracking  (SIFT)  algorithm  ([7]). 

Our  testbed  software  has  implemented  various  optical  flow  algorithms  including  block  matching  and  KLT  algorithms.  A 
central  feature  track  manager  is  created  to  tightly  coupled  optical  flow,  ASAF  and  SFM  to  provide  feature  image 
position  prediction.  6DOF  feedback  from  the  central  fusion  estimator  to  the  optical  flow  and  SFM  was  found  very  useful 
in  important  aircraft  maneuvers  as  shown  in  section  3.1. 

2.2  Structure  from  motion 

Structure  from  motion  (SFM)  is  responsible  for  combining  Optical  Flow  (OF)  processing  with  ASAF  information  in 
order  to  estimate  the  3D  positions  of  the  features  that  are  being  tracked  within  an  image.  SFM  can  be  thought  of  as 
somewhat  of  an  intermediary  between  OF  and  the  Navigation  system,  as  it  takes  input  from  and  provides  feedback  to 
both  systems,  improving  the  performance  of  each  as  shown  in  Figure  2. 


Figure  2:  SFM  combines  optical  flow  processing  with  vehicle  navigation 

The  3D  feature  positions  calculated  by  SFM  is  fed  back  into  both  the  OF  and  ASAF  modules.  3D  position  estimates  can 
be  used  to  predict  2D  image  locations  of  features,  aiding  OF  in  tracking  the  features.  Furthermore,  the  tracking  quality 
estimate  of  the  feature  can  be  used  to  aid  OF  in  culling  features  that  are  likely  being  mistracked,  freeing  up 
computational  resources.  While  to  some  degree  the  3D  location  of  features  are  already  being  computed  implicitly  within 
the  navigation  system,  and  as  such  SFM  is  not  directly  necessary,  the  explicit  SFM  system  provides  a  number  of 
additional  benefits  not  available  if  it  is  left  as  an  implicit  part  of  Navigation. 

First  of  all,  SFM  allows  for  an  enhanced  estimate  of  the  track-quality  of  a  given  feature.  Secondly,  an  explicit  SFM 
dramatically  increases  the  number  of  feature  positions  that  can  be  estimated  allowing  for  the  construction  of  much 
denser  maps.  Finally,  SFM  allows  for  a  more  convenient  and  safer  decoupling  of  features,  avoiding  the  potential  issues 
of  system  corruption  that  arise  due  to  feature  mistracking. 


Proc.  of  SPIE  Vol.  6575  657509-3 


2.3  All  source  adaptive  fusion  testebed 

Northrop  Grumman  and  AFRL  jointly,  developed  an  end-to-end  ASAF  testbed  environment  for  testing  the  overall 
system  performance  against  key  performance  parameters.  The  testbed  encompasses  a  6-DOF  simulation,  physics-based 
scene  generation  tools,  a  suite  of  optical  flow  software,  DTED,  GPS/INS,  and  baro-altimeters.  The  testbed  software  is 
modular,  and  can  be  reconfigured  to  work  in  real-time  mode  to  receive  real  imagery  and  navigation  data.  The  ASAF 
module  adaptively  fuses  in  real-time  all  available  navigation  data  such  as  IMU/GPS,  altimeters,  star  tracker,  passive 
imaging  sensor,  stored  imagery,  and  digital  elevation  database.  By  optimally  fusing  all  sources  of  data,  ASAF  can 
significantly  improve  navigation  and  targeting  performance  both  when  GPS  is  present  and  when  GPS  is  denied  as  shown 
in  section  3.  ASAF  software  module  is  hosted  on  a  Laptop  computer  and  is  a  part  of  a  flying  testbed  on  a  Cessna  Grand 
Caravan. 


3.  PERFORMANCE  RESULTS 

The  ASAF  has  been  tested  using  simulated  and  filed  data  as  described  in  the  following  paragraphs. 

3.1  Optical  flow  performance 

The  following  example  illustrates  the  optical  flow  performance.  The  moving  platform  moves  in  a  straight  flight  and 
turns  with  a  bank  angle  of  20  degree.  Terrain  images  are  generated  by  Irma  (multi-sensor  signature  prediction  code). 
Terrain  images  represent  a  rural  environment  with  random  bushes  and  sparse  buildings.  Based  on  Irma  outputs  and 
6DOF  simulation  parameters,  theoretical  optical  flow  and  selected  image  feature  position  can  be  computed.  We 
compared  two  optical  flow  algorithms  KLT  and  feedback  block  matching  as  shown  in  Figure  3.  When  aircraft  flies 
straight  without  acceleration,  the  performance  of  the  two  algorithms  are  comparable.  But  when  the  aircraft  turns,  the 
KLT  algorithm  fails  completely,  while  the  6DOF  feedback  continues  to  maintain  good  performance. 
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Figure  3.  Optical  flow  performance:  KLT  (dotted  line)  and  6DOF  feedback  block  matching  algorithm  (solid  line) 
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3.2  Structure  from  motion  performance 

SFM  performance  was  evaluated  as  shown  in  Figures  4  and  5.  These  results  are  based  on  a  simulated  vehicle  flying  a 
simple  trajectory  with  a  single  camera  looking  straight  down.  Optical  flow  measurements  are  generated  according  to  the 
truth  model  for  a  given  feature.  At  generation  each  feature  is  assigned  an  initial  true  location  in  space  and  a  random 
measurement  noise  value  used  to  corrupt  the  feature,  providing  a  reasonable  amount  of  variation  in  feature  quality. 
Since  the  image  projection  function  is  not  uniquely  invertible,  features  within  SFM  are  initialized  based  on  noisy  a  priori 
terrain  information.  Our  choice  of  parameters  in  this  simulation  resulted  in  an  initial  error  roughly  ranging  between  0 
and  80  meters  (although  occasionally  as  large  as  140  m).  Figure  4  (a)  demonstrates  that  SFM  successfully  combines  the 
noisy  2D  measurements  from  the  camera  into  a  refined  3D  estimate  of  location.  The  error  values  for  most  features  are 
nearly  monotonically  decreasing,  and  converge  correctly  to  the  true  positions.  It  is  evident  from  Figure  4  (a)  that  the 
error  is  less  than  20  meters  for  most  features  by  the  10th  frame.  Figure  4  (b)  and  Figure  5  additionally  show  the 
effectiveness  of  our  Track  Score  calculation.  We  can  differentiate  between  features  with  Track  Score  >  1  at  the  10th 
frame.  In  Figure  4  (b),  it  is  evident  that  the  features  with  higher  Track  Scores  have  substantially  less  error  than  those 
with  lower  Track  Scores.  Furthermore,  in  Figure  5,  it  is  evident  that  there  is  a  strong  correlation  between  the  Track 
Score  and  both  the  measurement  noise  and  the  initial  error.  By  choosing  features  with  higher  scores  for  use  with  the 
navigation  system  we  can  ensure  that  the  selected  features  have  lower  measurement  noise  and  error  values,  making  them 
far  more  beneficial  and  less  likely  to  corrupt  the  system. 

Error  evolution  of  features  Error  evolution  of  Features  by  T rack  Score 


Frame  Frame 


Figure  4.  (a)  Error  between  estimated  feature  position  and  true  position  vs.  frame  numbe,  (b)  Error  vs.  track  score  and 
frame  number 
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Meas.  noise  (std  dev  in  pixels)  Error  (m) 


Figure  5.  Computed  track  score,  measurement  noise  (standard  deviation  in  pixels  of  noise  added  to  the  simulated  true 
measurement),  and  initial  error  values 
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3.3  ASAF  navigation  performance 

ASAF  has  been  tested  using  both  simulated  and  real  data  yielding  promising  results.  It  is  evident  from  the  performance 
evaluation  that  ASAF  yields  an  improvement  over  GPS  only  solution  in  the  presence  of  GPS  and  provides  reliable 
navigation  aids  in  the  non-GPS  environment. 

The  following  example  illustrates  the  navigation  performance  for  a  navigation-grade  INS/GPS.  Figure  6  shows  the 
position,  heading,  and  velocity  errors  that  result  when  GPS  is  lost  at  t  =  300  sec.  Figure  7  shows  the  same  parameters, 
plotted  to  the  same  scale,  for  the  case  where  the  INS/GPS  data  are  optimally  fused  with  vision  data  starting  at  t  =  250 
sec.  The  imaging  sensor  is  tracking  five  ground  features  which  are  known  to  be  stationary,  but  whose  positions  are 
unknown. 
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Figure  6.  INS/GPS  performance 
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Figure  7  INS/GPS  performance  with  optical  flow  aiding 
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The  next  example  shows  the  improvement  in  EO/IR  target  location  accuracy  that  can  be  obtained  using  ASAF.  The 
FLIR  system  is  assumed  to  have  a  resolution  of  .02  mrad  and  mounting  tolerance  plus  gimbal  misalignment  of  1  mrad. 
The  INS/GPS  mounting  tolerance  and  uncompensated  structure  flexure  errors  are  assumed  to  sum  to  1  mrad.  A  laser 
range  finder  with  resolution  of  1  meter  is  assumed  available.  The  targeting  vehicle  is  flying  at  an  altitude  of  20,000  feet 
with  a  look-down  angle  of  45  degrees.  Figure  8  shows  the  target  location  errors  (TFE)  for  INS/GPS  only,  compared 
with  the  results  for  INS/GPS  plus  feature  tracking,  again  with  GPS  denied  at  t  =  300  sec.  Figure  9  shows  the  feature 
tracking  results  at  an  expanded  scale.  As  shown  in  Figure  10,  feature  tracking  improves  TFE  even  when  GPS  is 
continuously  available. 


Target  Location  Error 


Figure  8.  TLE  comparison  plots  -  GPS  lost  at  time  =  300  sec 


Target  Location  Error 


Figure  9.  TLE  expanded  scale  -  feature  tracking  on 
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Target  Location  Error 


Figure  10.  Comparison  of  TLE  with  and  without  feature  tracking  -  GPS  on 

The  preceding  simulation  results  were  obtained  by  fusing  vision  tracks  of  (unknown)  discrete  ground-fixed  features  with 
GPS/INS  data.  Table  1  shows  simulation  results  where  2-D  optical  flow  field  velocities,  rather  than  discrete  tracks,  are 
fused  with  GPS/INS  data.  Experiments  were  also  run  where  star  tracker  measurements  were  simulated  in  addition  to 
optical  flow  aiding.  The  star  tracker  modeled  was  a  relatively  low-precision,  low-cost,  miniature  star  tracker  (3  inches  x 
3  inches)  designed  by  Northrop  Grumman.  Two  grades  of  INS/GPS  were  simulated,  and  two  levels  of  optical  flow 
measurement  accuracy.  It  was  shown  that  by  fusing  optical  flow,  star-tracking  measurements  with  inertial  data, 
navigation  accuracy  in  a  GPS-denied  environment  is  obtained  that  is  superior  to  that  of  INS/GPS  alone  when  GPS  is 
available. 


Table  1.  Fusion  of  IMU,  GPS,  optical  flow,  and  star-tracker  data 


1  “Inertial”  =  Accels  +  Gyros  +  Altimeter 

2  “Stellar-Inertial”  =  Inertial  +  Star  Tracker  (10  arc-sec) 

3  Includes  gravity  data  base 

4  Registration  accuracy 

CEP 

Rate-Grade 

INS 

(LN200) 

Nav-Grade 

INS3 

(LN250) 

Mission 

Nav  Mode 

.1  pix4 

.3  pix4 

.1  pix4 

.3  pix4 

Smart  Munition 

-  Alt=1000  ft,  V=200  ft/sec 

-  S-curve  flight  path 

-  30  min  after  loss  of  GPS 

Inertial*: 
Stellar-Inertial2: 
Inertial  +  Optical  Flow: 
Stellar-Inertial  +  Op 
Flow: 

16,000 

500 

83 

17 

16,000 

500 

108 

21 

3,500 

138 

23 

13 

3,500 

138 

28 

16 

Fire  Scout 

-Alt=1 5,000  ft,  V=200  ft/sec 
- 18  mile  racetrack 
-  3  hrs  after  loss  of  GPS 

Inertial: 
Stellar-Inertial: 
Inertial  +  Optical  Flow: 
Stellar-Inertial  +  Op 
Flow: 

40,000 

310 

44 

30 

40,000 

310 

180 

43 

16,000 

303 

33 

21 

16,000 

303 

50 

32 

4.  CONCLUSIONS 

The  all  source  adaptive  fusion  has  been  implemented  to  fuse  in  real  time  any  available  information  to  provide  fully 
autonomous  robust  navigation  capability  in  GPS  jamming  or  signal  interruption  environment.  ASAF  is  implemented  in 
a  testbed  environment  to  support  sensitivity  analysis  and  performance  evaluation  in  simulated  environment  as  well  as  in 
flight  demonstration  and  testing  using  a  flying  Cessna  Grand  Caravan  with  mounted  cameras  and  navigation  equipment. 
ASAF  yields  superior  navigation  performance  and  can  be  used  to  reduce  target  location  error  and  provide  moving  target 
indication. 
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